Skip to content

channel-memory: async LLM digest worker with cost caps - #272

Merged
mostlydev merged 2 commits into
masterfrom
codex/issue-268-async-digest-worker
May 23, 2026
Merged

channel-memory: async LLM digest worker with cost caps#272
mostlydev merged 2 commits into
masterfrom
codex/issue-268-async-digest-worker

Conversation

@mostlydev

@mostlydev mostlydev commented May 22, 2026

Copy link
Copy Markdown
Owner

Summary

Adds the async, off-hot-path LLM digest worker to the channel-memory adapter — the final slice of the issue #232 Phase 2 / PR #262 design. It compresses verbose raw_excerpt windows into sparse topic_rollup/sequence_rollup blocks via an LLM, while /digest stays strictly LLM-free (reads already-generated blocks only).

Conservative by design:

  • only raw_excerpt windows are summarized; hard events, tombstones, telemetry keep their faithful deterministic blocks
  • requires structured JSON citing exact source message ids; malformed or provenance-free output is rejected and deterministic blocks keep serving
  • caches by source-message ids + content hashes (unchanged windows never re-summarized)
  • stores provider/model/version/cost per block in metadata_json
  • enforces conservative per-channel and per-pod daily call caps (usage tracked in a new llm_usage table); deterministic-only fallback when disabled, over budget, or failing
  • editing/deleting/forgetting a covered source dirties the rollup via shared provenance, so stale summaries stop serving
  • disabled unless CHANNEL_MEMORY_LLM_ENABLED=true + an OpenAI-compatible CHANNEL_MEMORY_LLM_BASE_URL are set

Acceptance coverage (tests)

  • verbose→sparse compression with faithful hard events + provenance
  • queue ordering (oldest window first under a cap)
  • malformed-output + provenance-free + hallucinated-source rejection
  • cache reuse by content hash
  • per-pod cost-cap enforcement
  • deterministic fallback when disabled / when the model errors
  • edit + forget provenance invalidation (synthetic source events — claw-wall observes first-sightings only)
  • no LLM call in the /digest hot path

Validation

Closes #268
Refs #262

@mostlydev
channel-memory: add async LLM digest worker with cost caps (#268)
748c6c5 
Adds an off-hot-path background worker to the channel-memory adapter that
compresses verbose raw_excerpt windows into sparse topic_rollup /
sequence_rollup blocks via an LLM. /digest stays LLM-free — it only reads
already-generated blocks.

The worker is conservative: only raw_excerpt windows are summarized (hard
events, tombstones, telemetry keep their faithful deterministic blocks);
results must be structured JSON citing the exact source message ids, and
malformed or provenance-free output is rejected; work is cached by source
ids + content hashes; each block records provider/model/version/cost in
metadata_json; per-channel and per-pod daily call caps are enforced (usage
tracked in a new llm_usage table) with deterministic-only fallback when
disabled, over budget, or failing. Editing/deleting/forgetting a covered
source dirties the rollup through shared provenance so stale summaries stop
serving. The worker is disabled unless CHANNEL_MEMORY_LLM_ENABLED=true and
an OpenAI-compatible CHANNEL_MEMORY_LLM_BASE_URL are configured.

Tests cover verbose->sparse compression with faithful hard events, queue
ordering, malformed/provenance-free rejection, cache reuse by content hash,
per-pod cost-cap enforcement, deterministic fallback when disabled/failing,
and edit/forget provenance invalidation (synthetic source events).

Closes #268
Refs #262
@mostlydev

mostlydev commented May 23, 2026

Copy link
Copy Markdown
Owner Author

Codex independent review of PR #272 at 748c6c5:

Findings:

  1. writeSparseRollup stales the deterministic raw_excerpt blocks it replaces, but the invalidation path only dirties the LLM rollup. When a covered source is later edited, deleted, or forgotten, /digest stops serving the dirty LLM block, but the unchanged current source messages covered by that rollup remain stale and are not restored as deterministic fallback. That drops channel context until the worker successfully regenerates the rollup, which violates the deterministic fallback and edit/delete/forget acceptance criteria.

  2. channel-memory: add async LLM digest worker and cost caps #268 calls for conservative per-channel/per-pod daily call and cost caps. The PR tracks cost_usd, but the actual budget checks only cap call counts; there is no USD cap config or pre-call cost-budget check, so an over-cost channel/pod cannot fall back to deterministic-only output.

Focused local review note: the active workspace owner has unpushed in-progress fixes for both issues, but the PR head is still 748c6c5, so this comment is against the code currently on GitHub.

@mostlydev

mostlydev commented May 23, 2026

Copy link
Copy Markdown
Owner Author

Independent review found two blockers in the initial async worker slice: rollup invalidation could suppress deterministic raw fallback for still-current messages, and the worker claimed cost caps while enforcing only call caps. Pushed a follow-up fix in a338f8f.

Fixes included:

  • non-destructive sparse-rollup preference at /digest serve time, so raw_excerpt blocks remain available whenever a rollup is dirty/stale
  • full-window provenance validation; partial citations are rejected before any sparse block is written
  • attempted calls count against daily call caps, including malformed/rejected outputs
  • per-channel and per-pod USD budget knobs/enforcement, with estimated per-call cost fallback
  • numeric OpenAI-compatible temperature and a model/base-url guard for enabling the worker

Validation:

  • go test ./...
  • go test -count=1 ./examples/channel-memory/
  • go test ./cmd/claw-wall ./internal/pod
  • go vet ./...
  • go build ./...
  • git diff --check

No remaining blockers from my review.

@mostlydev
mostlydev merged commit b4f8683 into master May 23, 2026
@mostlydev
mostlydev deleted the codex/issue-268-async-digest-worker branch May 23, 2026 17:38
@mostlydev

mostlydev commented May 23, 2026

Copy link
Copy Markdown
Owner Author

Codex re-review after hardening commit a338f8f / merge b4f8683:

The two findings in my earlier review comment are addressed. The merged code now keeps deterministic raw blocks available as fallback while preferring fresh sparse rollups at serve time, restores fallback coverage when rollups are dirtied, validates full-window provenance, counts attempted calls against caps, and adds per-channel/per-pod USD budget knobs/enforcement.

Verification I ran locally on the pushed head before merge:

  • go test -count=1 ./examples/channel-memory/
  • go test ./...
  • go vet ./...
  • go build ./...
  • git diff --check origin/master...HEAD

No remaining blockers from my review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

channel-memory: add async LLM digest worker and cost caps

1 participant

@mostlydev